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A sequential transducer, derived from a Hidden Markov Model, that closely approximates the behavior of the stochastic model. The 
invention provides (a) a method (called n-type approximation) of deriving a simple finite-date transducer which is applicable in all cases, 
from HMM probability matrices, (b) a method (called s-type approximation) for building a precise HMM transducer for selected cases 
which are taken from a training corpus, (c) a method for completing the precise (s-type) transducer with sequences from the simple (n-type) 
transducer, which makes the precise transducer applicable in all cases, and (d) a method (called b-type approximation) for braiding an 
HMM transducer with variable precision which is applicable in all cases. This transformation is especially advantageous for part-of-speech 
tagging because the resulting transducer can be composed with other transducers that encode correction rules for the most frequent tagging 
errors. The speed of tagging is also improved. The described methods have been implemented and successfully tested on six languages. 
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1 

FSTs approximating Hidden Markov Models and text tagging using same 

The present invention relates to computer-based text processing, and more particularly to 
techniques for part-of-speech tagging by finite state transducer (FST) derived from a Hidden 
5 Markov Model (HMM). 

Part-of-speech tagging of machine-readable text, whereby tags are applied to words in a 
sentence so as to identify a word's part-of-speech (e.g. noun-singular, verb-presenMst person 
plural), are known. Such tags are typically of a standardised form, such as specified under the 
Text Encoding Initiative (TE1). A text or corpus, once tagged, finds use, for example, in 
10 information retrieval from the text and statistical analysis of the text 

Probabilistic HMM based taggers are known from Bah! and Mercer (1976) and Church 
(1988); see Section H: References at the end of this disclosure. They select among the part-of- 
speech tags that are found for a word in the dictionary, the most probable tag based on the word's 
context, i.e. based on the adjacent words. 
15 However, a problem with a conventional HMM based tagger is that it cannot be integrated 

with other finite state tools which perform other steps of text analysis/manipulation. 

One method is to generate an FST performing part-of-speech tagging from a set of rules 
written by linguists (Chanod and Tapanainen, 1995). This, however, takes considerable time and 
produces very large automata (or sets of automata that cannot be integrated with each other for 
20 reasons of size) which perform the tagging relatively slowly. Such an FST may also be non- 
deterministic, i.e. for an input sentence it may provide more than one solution or no solutions at 

an. 

The present invention provides methods to approximate a Hidden Markov Model (HMM) 
used for part-of-speech tagging, by a finite-state transducer. (An identical model of an HMM by a 
25 transducer (without weights) is in many cases impossible.) In specific embodiments there are 
provided (a) a method (called n-type approximation) of deriving a simple finite-state transducer 
which is applicable in all cases, from HMM probability matrices, (b) a method (called s-type 
approximation) for building a precise HMM transducer for selected cases which are taken from a 
training corpus, (c) a method for completing the precise (s-type) transducer with sequences from 
30 the simple (n-type) transducer, which makes the precise transducer applicable in all cases, and 
(d) a method (called b-type approximation) for building an HMM transducer with variable precision 
which is applicable in all cases. 

The invention provides a method of generating a text tagging FST according to any of 
claims 1, 5 and 9 of the appended claims, or according to any of the particular embodiments 
35 described herein. 

The invention further provides a method of generating a composite finite state transducer 
by composing the HMM-derived FST with one or more further text-manipulating FSTs. 

The invention further provides a method of tagging a text using the aforementioned HMM- 
derived FST or the composite FST. 
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The invention further p novices a text processing syjtem according to aakn t1 nf the 
appended claims, and a recordable medium according to claim 12 of the appended claims. 

The HMM transducer builds on the data (probability matrices) of the underlying HMNL The 
accuracy of the data collected in the HMM training process has an impact on the tagging accuracy 
5 of both the HMM itself and the derived transducer. The training of this HMM can be done on either 
a tagged or untagged corpus, and is not discussed in detail herein since it is exhaustively 
described in the literature (Bahl and Mercer, 1976; Church, 1988). 

An advantage of Finite-State taggers according to the invention is that tagging speed 
when using transducers is up to five times higher than when using the underlying HMM (cf. 
10 section F hereinbelow). 

However, a main advantage of the invention is that integration with tools performing 
further steps of text analysis is possible: transforming an HMM into a FST means that this 
transducer can be handled by the finite state calculus and therefore cfirectly integrated into other 
finite-state text processing tools, such as those available from XEROX Corp., and elsewhere. 
15 Since the tagger is in the middle of a chain of text analysis topis where all the other tools may be 
finite-state-based (which is the case with text processing tools available from XEROX Corp.), 
converting the HMM into an FST makes this chain homogeneous, and thus enables merging the 
chain's components into one single FST, by means of composition. 

In particular, it is possible to compose the HMM-derived transducer with, among others, 
20 one or more of the following transducers that encode: 

• correction rules for the most frequent tagging errors in order to significantly improve tagging 
accuracy. These rules can either be extracted automatically from a corpus (Brill, 1992) or 
written manually (Chanod and Tapanainen. 1995)i The rules may include long-distance 
dependencies which are usually not handled by HMM taggers. 

25 • further steps of text analysis, e.g. light parsing or extraction of noun phrases or other phrases 
(Ait-Mokhtar and Chanod. 1997). 

• criteria which decide on the relevance of a corpus with respect to a particular query in 
information retrieval (e.g. occurrence of particular words in particular syntactic structures). 

These transducers can be composed separately or ail at once (Kaplan and Kay, 1994). Such 
30 composition enables complex text analysis to be performed by a single transducer see EP-A- 
583,083. 

It w3J be appreciated that the algorithms descrfced herein may find uses beyond those 
discussed with respect to the particular embodiments discussed below: i.e. on any land of analysis 
of written or spoken language based on both finite-state technology and HMMs, such as corpus 
35 analysis, speech recognition, etc. The algorithms have been fuDy implemented. 

Embodiments of the invention will now be described, by way of example, with reference to 
the accompanying drawings, in which: 

Figure 1 illustrates decisions on tags with an n-type transducer, accocding to one 
embodiment of the invention; 
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Figure 2 illustrates schematically the gene.-atiun Oi" an n-tyoe transducer in accordance 
with an embodiment of the invention; 

Figure 3 illustrates the procedure of building an n-type transducer in accordance with an 
embodiment cf the invention; 
5 Figure 4 is a diagram illustrating class subsequences of a sentence; 

Figure 5 is a schematic diagram of the steps in the procedure of building an s-type 
transducer, in accordance with an alternative embodiment of the invention; 

Figure 6 is a diagram illustrating the disambiguation of classes between two selected 

tags; 

10 Figure 7 is a diagram illustrating valid paths through the tag space of a sentence. 

Figure 8 is a diagram illustrating b-type sequences. 

Figure 9 is a schematic flow chart illustrating the procedure of building a b-type 
transducer 

Figure 10 is an illustration of the tagging of a sentence using either the n-type transducer 
15 formed in Fig. 3 or the s-type transducer formed in Ftg. 5 or the b-type transducer formed in Fig. 
9; and 

Figure 11 is a schematic flow chart of the steps involved in the procedure, in accordance 
with an embodiment of the invention, of tagging a text corpus with a finite state tagger using an 
HMM transducer. 

20 

A. System configuration 

It will be appreciated that the techniques according to the invention may be employed 
using conventional computer technology. It will be appreciated that the invention may be 
implemented using a PC running Windows™, a Mac running MacOS, or a minicomputer running 
25 UNIX, which are well known in the art For example, the PC hardware configuration is discussed 
in detail in The Art of Electronics, 2nd Edn, Ch. 10, P. Horowitz and W. Hill, Cambridge University 
Press, 1989. The invention has been implemented in C on a Sun Sparc 20 workstation running 
UNDC 

30 B. FST derivation 

The invention wOl be descrbed in by reference to three methods of deriving a transducer 
for part-of-speech tagging from an HMM. These methods and transducers are referred to herein 
as n-type, s-type and b-type. 

An HMM used for tagging encodes, like a transducer, a relation between two languages. 
35 One language contains sequences of ambiguity classes obtained by looking up in a lexicon all the 
words of a sentence. The other language contains sequences of tags obtained by statistically 
disambiguating the class sequences. From outside, an HMM tagger behaves like a sequential 
transducer that deterministically maps every sequence of ambiguity classes (corresponding to a 
sentence) into a unique sequence of tags, e.g.: 



SUBSTITUTE SHEET (RULE 26) 



WO 99/01828 



PCI7EP98/04153 



4 

PET] [ADJ.KMJI^ [ADJ,NOUfi] [END] 

DET ADJ NOUN END 

5 C. n-Type Transducers 

This section presents a method that approximates a first order Hidden Markov Model 
(HMM) by a finite state transducer (FST), referred to as an n-type approximation. Figure 1 
illustrates decisions on tags with an n-type transducer, according to one embodiment of the 
invention. 

10 As in a first order HMM we take into account initial probabilities x, transition probabilities a 

and class (Le. observation symbol) probabilities b. However, probabilities over paths are not 
estimated Unlike in an HMM, once a decision is made, it influences the following decisions but is 
itself irreversible. 

Figure 1 illustrates mis behaviour with an example: for the class cf of word wy, tag ti 2 is 
15 selected which is the most iceiy tag in an initial position. For word w^, the tag (22. the most likely 
tag, is selected given class c£ and the previously selected tag ti2, etc 

A transducer encocfng this behaviour can be generated as illustrated in Fig. 2: this shows 
schematically an rvtype transducer in accordance with an embocfiment of the invention. In this 

example there is a set of three classes, cy with the two tags f/f and tf2, C2 with the three tags 

20 t21, t22 and (23. and 03 with one tag (37. Different classes may contain the same tag, e.g. ti2 

and t23 may refer to the same tag (e.g. [NOUN]). Figure 3 Sustrates the procedure of building an 
n-type transducer in accordance with an embodiment of the invention. 

Starting with the set of tags and the set of classes (generally designated 2), for every 

possible pair of a class and a tag (e.g. 0^2 or [ADJ,NOUN]:NOUN) a unique state is created 
25 (step s1) and labelled with this same pair. This set of states will aOow to map any class to anyone 
of its tags. An initial state which does not correspond with any pair, is also created in step s1. AO 
states are final, marked by double circles. This produces a set of states labelled with class-tag 
pairs and 1 initial state (generally designated 4). 

For every state, as many outgoing arcs are created (step s2) as there are classes (three 
30 in Fig. 2). Each such arc for a particular class points to the most probable pair of this same class. 
The set of outgoing arcs of one state will allow to decide on the following tag, based on the 
following class and the current state's tag. If the arc comes from the initial state, the most 
probable pair of a class and a tag (destination state) is estimated by the initial and class 
probability of the tag: 

35 argmax pfotj = n (f tt ) Kc^) 
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If the arc comes from a state other ttuji the initial state, the <nost prooable pair is esxinmtod by the 
transition and class probability of the tag: 

argmax p 2 (c i ,t ik ) = a^t previo J ^(c^ ) 

In the example (Fig. 2), C1^12 fe the most likely pair of class c-|, and C2^23 ^© most 
5 likely pair of class C2 when coming from the initial state, and C2^21 the most likely pair of class C2 

when coming from the state of 03^31 . 

Every arc is labelled with the same symbol pair as its destination state, with the dass 
symbol in the upper language and the tag symbol in the lower language. Eg. every arc leading to 

the state of ct ^12 is labelled with d *12- The result is the non-minimal and non-deterministic FST 
10 6. 

Finally, all state labels can be deleted since the behaviour described above is encoded in 
the arc labels and the network structure. The network can be minimised and determinised (step 
s3) to produce the n-type FST 8. 

The above mathematical model is referred to as an nl-type model, the resulting finite- 
15 state transducer an n1-type transducer and the whole algorithm leading from the HMM to this 
transducer, an n1~type approximation of a first order HMM. 

Adapted to a second order HMM, this algorithm would give an n2-type approximation. 
Every n-type transducer is sequential, Le. deterministic on the input side. In every state 
there is exactly one outgoing arc for every class symbol An n-type transducer tags any corpus 
20 deterministtcally. 

D. s-Type Transducers 

This section presents a method, according to another embodiment of the invention, that 
approximates a first order Hidden Markov Model (HMM) by a finite state transducer (FST), 
25 referred to herein as an s-type approximation. 

D.1 Mathematical Background 

To tag a sentence Le. to map its class sequence to the most probable tag sequence, me 
can split the class sequence at the unambiguous classes (containing one tag only) into 
30 subsequences, then tag them separately and concatenate them again. The result is equivalent to 
the one obtained by tagging the sentence as a whole. 

Figure 4 is a diagram illustrating class subsequences of a sentence. Two types of 
subsequences of classes are distinguished: initial and middle ones. The final subsequence of a 

sentence is equivalent to a middle one, if it is assumed that the sentence end symbol (. or ! or ?) 
35 always corresponds to an unambiguous class Cy. 
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An initial subsequence Cj starts with the sentence inrtiai position rr\ has any numbsr 

(inci. zero) of ambiguous classes and ends with the first unambiguous class Cq of the sentence. It 
can be described by the regular expression: 

5 Given an initial class subsequence C\ of length r, its joint probability together with a 

corresponding initial tag subsequence 7/ can be estimated by: 



p(Q y Tl) = x(t l )b(c l \t l ) 



r-1 



U=2 



10 A middle subsequence Cm starts immediately after an unambiguous class has any 

number (incl. zero) of ambiguous classes c a and ends with the following unambiguous class cu- It 
can be described by the regular expression: 

To estimate the probability of middle tag sequences Tm correctly, we have to include the 

0 v 

15 immediately preceding unambiguous class Cu actually belonging to the preceding subsequence 
C/or Cm Thus we obtain ah extended middle subsequence: 
wn — c u c b* c u 

e 

The joint probability of an extended middle class subsequence Cm of length s together 

e 

with a comesponcfing tag subsequence Tm can be estimated by 

s-l 



20 
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DJZ Building an a-type transducer 

To build an s-type transducer, we generate a large number of initial class sequences C; 

o 

and extended middle class sequences C m as described in D.3 below, and disambiguate (i.e. 

tag) each of them based on a first order HMM using the Vrterbi algorithm for efficiency (Viterbi, 
5 1967; Rabiner, 1990). 

Every class subsequence gets finked to its most probable tag subsequence by means of 
the cross product operation: 

S/=C/ jc Tj=sci:ti czt2 — Cp:t n 

Sm e = C m e x.T m e =ci e :ti e cz t2 — ctfto 

U U e 

10 Then the union S/ of all initial subsequences Sj and the union S m of all extended middle 

0 

subsequences Sm is built 

e 

In all extended middle subsequences Sm . like e.g. 

C m 9 [DET][AE>J,NOUhq[ADJ,NOUNIINOUN] 

Sm = = " 

15 Tm DET ADJ ADJ NOUN 

the first class symbol on the upper side and the first tag symbol on the lower side will be marked 
as an extension that does not realty belong to the middle sequence but is necessary to 
disambiguate it correctly. The above example becomes 
0 

20 C m 0.[DET][ADJ f NOUN][ADJ,NOyr4][NOUr4] 

S m = = 
0 

T m O.DET ADJ ADJ NOUN 

Now it is possible to formulate a preliminary sentence model containing one initial subsequence 
25 foiowed by any number find, zoo) of extended middle subsequences: 

UO 17 1/ 0 

S o Si S m * 

0 

where all middle subsequences S m are stfli marked and extended in the sense that all 

occurrences of all unambiguous classes are mentioned twice: Once at the end of every sequence 
(unmarked) and also at the beginning of every following sequence (marked). 
30 To ensure a correct concatenation of initial and middle subsequences a concatenation 

constraint for classes is formulated: 
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staling that every middle subsequence must begin with the same n<arked unrmbinuous dass cj 

(e.g. 0.[DE7D that occurs unmarked as c u (e.g. [DETD at the end of the preceding subsequence 

since both symbols refer to the same occurrence of this unambiguous class. 

Having ensured correct concatenation by composition of the preliminary sentence model 

UO 

5 S with the concatenation constraint Rc, all marked classes on the upper side and all marked 
tags on the lower side of the relation are deleted. 

The above mathematical model is referred to as an s-type model, the corresponding 
finite-state transducer an s-type transducer and the whole algorithm leading from the HMM to the 
transducer, an s-type approximation of an HMM. 

10 An s-type transducer tags any corpus that does not contain unknown subsequences 

exactly the same way. Le. with the same errors, as the corresponding HMM tagger does. An s- 
type transducer is, however, incomplete because it encodes only a limited set of subsequences of 
ambiguity classes. Therefore an s-type transducer cannot tag sentences with one or more 
subsequences that are not in this set An s-type transducer can be completed as deserted in D.4 

15 below. 

03 Generation of ambiguity dass subsequences 

This section describes three ways to obtain class subsequences that are needed to buBd 
an s-type transducer. Figure 5 is a schematic diagram of the steps in the procedure of fauOding an 
20 s-type transducer, in accordance with an alternative embodiment of the invention. 

(a) Extraction from a corpus 

Based on a lexicon and a guesser an untagged training corpus (10) is annotated with 
class labels, exactly the same way as is done later for the^purpose of tagging. 

From every sentence we extract (step s4) the initial class subsequence C/that ends with 
25 the first unambiguous class cu, and all extended middle subsequences C m ° ranging from any 

unambiguous class c u in the sentence to the following unambiguous class. This generates the 

incomplete set of class subsequences (s-type), designated 12. 

(b) Generation of all possible subsequences 

Here, the set of tags and set of classes (generally designated 2) is used as the starting 

30 point The set of all classes c is split into the subset of unambiguous classes c u and the subset of 

ambiguous classes Cq. Then all possible initial and extended middle class subsequences, Cj and 
e 

C m up to a defined length are generated (step s5). 

As in D.3(a), an incomplete set of class sequences (s-type) 12 is obtained, and the HMM 
can be used (step s7) to obtain from this an incomplete set of class and tag sequences (s-type) 
35 14. This set 14 can be used with finite state calculus to build (step s8) an s-type transducer 16; 
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and this incomplete s-type FST in turn is used to derive the incomplete set 1 i, through extnxtisr. 
(step s10) of all class sequences, 
(c) Extraction from a transducer 

if an n-fype transducer N approximating an HMM is already available, initial and extended 

e 

5 middle class and tag subsequences, S/ and S m can be extracted (step s6) using finite state 
calculus, to produce a complete set 17 of class and tag sequences (n-type). 

D. 4 Completion of s-type transducers 

s-Type transducers with dass subsequences that have been generated as described in 
10 03(a) or 0.3(b) above, are in general not complete on the input side. They do not accept afl 
possible ambiguity class sequences. This, however, is necessary for a transducer used to 
disambiguate class sequences of any corpus since a new corpus can contain sequences not 
encountered in the training corpus. 

An incomplete s-type transducer S can be completed (step s12) with subsequences from 
15 an auxiliary complete n-type transducer N as follows: 

U U e 

Fast, the unions of initial and extended middle subsequences, 5S/ and s$m are 

U U e 

extracted from the primary s-type transducer S, and the unions nSy and n$m are extracted 

from the auxiliary n-type transducer N t as described in section D.3(c) above. 

U 

Then a joint union S/of initial subsequences is made: 
U U U u U 

U e 

and a joint union Sm of extended middle subsequences: 

U e U e u e U e U e 

In both cases all subsequences from the principal model S are unioned with afl those 
subsequences from the auxiliary model N that are not in SL 
25 Finally, the complete s+n-type transducer 18 is generated (step s14) from the joint unions 

U U 0 

of subsequences S/and Sm , as described in section DJZ above. 

A transducer completed in this way disambiguates all subsequences known to the 
principal incomplete s-type model exactly as the underlying HMM does, and all other 
subsequences as the auxiliary n-type model does. 

30 

E. b-Type Transducers 

This section presents a method, according to another embodiment of the invention, that 
approximates a first order Hidden Markov Model (HMM) by a finite-state transducer (FST), 
referred to herein as b-type approximation. Regular expression operators used in this section are 
35 explained in the annex. 
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E.1 Basic Idea 

Tagging of a sentence based on a (first order) HMM includes finding the most probable 
tag sequence given the class sequence of the sentence. 
5 In this approach, an ambiguity class is disambiguated with respect to a context A 

context consists of a sequence of ambiguity classes limited at both ends by some selected 
tag. For the left context of length p we use the term look-back, and for the right context of 

length a we use the term look-ahead. 

2 

In Fig. 6, the tag f /can be selected from the class c\ because it is between two selected 

1 2 
10 tags which are t i-2 at a look-back cfistance of p=2 and t j+2 at a look-ahead distance of ct=2. 

12 

Actually, the two selected tags f /-2and t /+£ allow not only the disambiguation of the dass cf but 

of all classes inbetween, i.e. c/.y, c/and c/+f. 

We approximate the tagging of a whole sentence by tagging sub-sequences with selected 
tags at both ends (Fig. 6), and then overlapping them. The most probable paths in the tag space 
15 of a sentence, i.e. valid paths according to this approach, can be found as sketched in Fig. 7. An 
ordered set of overlapping sequences where every sequence is shifted by one tag to the right with 
respect to the previous sequence, constitutes a valid path. There can be more than one valid path 
in the tag space of a sentence (Fig. 7). Sets of sequences that do not overlap in such a way are 
incompatfote according to this model, and do not constitute valid paths. 

20 

EJ2 b-Type Sequences 

Given a length p of look-back and a length a of lopk-ahead, we generate for every class 

co, every look-back sequence t^ c_^. , ... c_ 1# and every look-ahead sequence c x ... c^ t a , a b- 
type sequence: 

For example: 

CONJ [DET.PRON1 [ADJ.NOUN.VERB] [NOUN.VERB] VERB 

Each such original b-type sequence (Fig. 8) is disambiguated based on a first order HMM. 
Here we use the Viterbi algorithm (Viterbi, 1967; Rabiner, 1990) for efficiency. 
30 The algorithm will be explained for a first order HMM. In the case of a second order HMM, 

b-type sequences must begin and end with two selected tags rather than one. 

For an original b-type sequence, the joint probability of its class sequence C with its tag 
sequence 7 (Fig. 8), can be estimated by: 
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A boundary, i.e. a sentence beginning or end, may occur position in the look-back 
sequence and in the look-ahead sequence. No look-back (0=0) or no look-ahead (oc=0) is also 
allowed. The above probability estimation can then be expressed more generally (Fig. 8) as: 

5 PtP^^Pzart'PniMe'Perd 

with being 

P start =7l (t-fhl) 

Pttart ~ 1 
10 with Pmiddle being 



for fixed tag t_p 

for sentence beginning # 
for p =0, i.e. no look-back 



PnuddU =b (Co\t 0 ) 



for{tax>0 
forp+a=0 



and with pend being 

15 Pend^^a^a-D ^ for fixed tag f ff 

Paul = 1 for sentence end # or a= 0, Le. no look-ahead 

When the most likely tag sequence is found for an original b-type sequence, the class co 

in the middle position is associated with its most likely tag tQ. We formulate constraints for the 

other tags and t a and classes c_^ 7 ... c_ ; and c t of the original b-type sequence. Thus 
20 we obtain a tagged b-type sequence: 

... c^ corto Cj ... c^, t a 

stating that to is the most probable tag in the class qj if it is preceded by ... c_ ; and 

followed by c t ~. 
In the example: 
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CONJ-B2 [DET,PRON}-B1 [ADJ f NOUU,VEP.B]:ACV [NOUN,VERB}-A1 VEHB-A2 
ADJ is the most likely tag in the class [ADJ,NOUN,VERB] if it is preceded by the tag CONJ two 
positions back (B2), by the class [DET t PRON] one position back (B1), and followed by the class 
[NOUN.VERB] one position ahead (A1) and by the tag VERB two positions ahead (A2). 

5 Boundaries are denoted by a particular symbol #, and can occur anywhere in the look-back and 
look-ahead For example: 

#-B2 [DET.PRONJ-B1 [ADJ,NOUN,VERB]:ADJ (NOUN,VERB]-A1 VERB-A2 
CONJ-B2 [DET,PRON]-B1 [ADJ,NOUN,VERB]:NOUN #-A1 

Note that look-back length f) and look-ahead length a also include all sequences shorter 
10 than flora, respectively, that are limited by a boundary #. 

Figure 9 is a schematic diagram illustrating the steps in the procedure of building a b-type 
transducer, according to an embodiment of the invention. 

For a given length P of look-back and a length a of look-ahead, and using the set of tags 
and the set of classes, every possible original b-type sequence is generated (step s90). Then, 

15 these are disambiguated statistically, and encoded it in a tagged b-type sequence Bj as described 

above (Vrterbi; step s92). All sequences Bj are then untoned and a preliminary tagger model B* 
generated (step s94): 



B' = 



\)B. 



where all sequences Bj can occur in any order and number (including zero times) because no 
20 constraints have yet been applied. 

B3 Concatenation Constraints 

To ensure a correct concatenation of sequences, it is necessary to make sure that every 

sequence 8/ is preceded and followed by other 0/ according to what is encoded in the look-back 
25 and look-ahead which were explained above. 

Constraints are created for preceding and following tags, classes and sentence 
boundaries. For the look-back, a particular tag tj or class q is requested for a particular distance 
of fe-1, by: 

so tfic^i-fPcjlTc}* Tc |\ u c]W-l) c*^?*] 
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for&S-l 

u u 

with t and c being the union of ail tags and ail classes respectively. A sentence beginning, #, is 
requested for a particular look-back distance of 6<-l f by: 

forfe-1 

In the case of look-ahead, for a particular distance of fel, a particular tag tj or class cj or a 
sentence end # is required in a similar way, by: 

for&l 

We create the intersection f?f of all tag constraints R (tj), the intersection He of all class 

5 s 
constraints R (cp, and the intersection R#oi all sentence boundary constraints R 



15 



^=ry(#> 



All constraints (for tags r classes and sentence boundaries) are enforced (step s96 in Fig. 
9) by composition with the preliminary tagger model B*. The class constraint Rq must be 

composed on the upper stde of B' which is the side of the classes, and both the tag constraint Rf 

20 and the boundary constraint f?# must be composed on the lower side of B' which is the side of the 
tags: 

B m = R c .o. B' .o. f?f R# 
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Having ensured correct concatenation, all symbols that have served to constrain &gs, 
classes or boundaries are deleted (from B*)(step s98). Finally, the FST is determinized and 
minimized. 

5 The above-mentioned mathematical model is referred to as a b-type model, the 

corresponding FST as a b-type transducer, and the whole algorithm leading from the HMM to the 
transducer, as a b-type approximation of an HMM. 

E4 Properties of b-Type Transducers 

10 There are two groups of b-type transducers with different properties: FSTs without look- 

back or without look-ahead and FSTs with both look-back and look-ahead. Both accept any 
sequence of ambiguity classes. 

b-Type FSTs without look-back or without look-ahead are always sequential. They map a 
class sequence that corresponds to the word sequence of a sentence, always to exactly one 

15 tag sequence. Their tagging accuracy and similarity with the underlying HMM increases 
with growing look-back or look-ahead. A b-type FST with look-back p=l and without look- 
ahead (oe=0) is equivalent to an n1-type FST (section C). 

b-Type transducers with both look-back and look-ahead are in general not sequential. For 
a class sequence corresponding to the word sequence of a sentence, they deliver a set of 

20 alternative tag sequences, which means that the tagging results are ambiguous. This set is never 
empty, and the most probable tag sequence according to the underlying HMM is always in this 
set The longer the look-back distance p and the look-ahead distance a are, the larger the 
FST and the smaller the set of alternative tagging results. For sufficiently large look-back 
plus look-ahead* this set may contain always only one tagging result In this case the b- 

25 type FST is equivalent to the underlying HMM. For reasons of size however, this FST is only 
computable for HMMs with small tag sets. 

F. An Implemented Finite State Tagger 

The implemented tagger requires three transducers which represent a lexicon, a guesser 
30 and an approximation of an HMM. Figure 10 is an illustration of the tagging of a sentence using 
either the n-type transducer of Rg. 3, the s-type transducer formed in Fig. 5 or the b-type 
transducer formed in Fig. 9. Figure 11 is a schematic flow chart of the steps involved in the 
procedure, in accordance with an embodiment of the invention, of tagging a text corpus 20 with a 
finite state tagger using an HMM transducer. 
35 Every word 21 of an input sentence is read (step s71) from the corpus and is initially 

looked for in the lexicon; and if this fails, the search continues in the guesser (step s72), resulting 
in a word labelled with a class (22). As soon as an input token gets labelled with the sentence end 
class (e.g. [SENT] in Rg. 10), the tagger stops (step s73) reading words from the input At that 
point the tagger has read and stored the words of a whole sentence (Fig. 6, col. 1) and generated 
40 the corresponding sequence of classes 23 (see Rg. 10, coL 2). 
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The dass sequence 23 is now determmisticaKy mappod (s*ep s74) to a tag sequence 24 
(see Fig. 10 f col. 3) by means of the HMM transducer. 

The tagger outputs (step s75) the stored word and tag sequence 24 of the sentence and 
continues (step s76) the same way with the remaining sentences of the corpus, stopping (step 
5 s77) when the end of the corpus is reached. The end result is the tagged text corpus 25. 

G. Tests and Results 

Table 1 compares an n-type and an s-type transducer with the underlying HMM on an 
English test case. As expected, the transducers perform tagging faster than the HMM. - 

10 





tagging 

accuracy 

in% 


tagging speed on different 
computers in words/sec 


transducer size 


on ULTRA2 


on SPARC20 


num. stales 


num. arcs 


HMM 


96.77 


4590 


1564 


none 


none 


s-type transducer 


95.05 


12 038 


5453 


4709 


976785 


n-type transducer 


94.19 


17244 


8180 


71 


21087 



Language: 


English 


HMM training corpus: 


19 944 words 


Test corpus: 


19 934 words 


Tag set 


74 tags 297 classes 


s-Type transducer 


with subsequences from a training corpus of 100,000 words 
completed with subsequences from an n-type transducer 



Table 1 : Accuracy, speed, size and creation time of HMM transducers 



15 Since both transducers are approximations of HMMs, they show a sOghtiy lower tagging 

accuracy than the HMMs. However, improvement in accuracy can be expected since these 
transducers can be composed with transducers encoring correction rules for frequent errors, as 
descrfoed above in the introductory part of this disclosure. 

The s-type transducer is more accurate in tagging but also larger and slower than the n- 

20 type transducer. 

Table 2 compares the tagging accuracy of n-type and s-type transducer and the 
underlying HMM for different languages. 
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Tagging accuracy in % 


English 


Dutch 


French 


German 


Portug. 


Spanish 


HMM 


96.77 


94.76 


96.65 


97.62 


97.12 


97.60 


s-type transducer 


95.05 


92J36 


98.37 


95.81 


96.56 


96.87 


n-type transducer 


94.19 


91.58 


98.18 


94.49 


96.19 


96.46 



s-Type transducer with subsequences from a training corpus of 1 00,000 words 
completed with subsequences from n-type transducer 



Table 2: Accuracy of HMM transducers for different languages 

5 

For the test of b-type transducers we used an English corpus, lexicon and guesser. that 
originally were annotated with 74 different tags. We automatically recoded the tags in order to 
reduce their number, i.e. in some cases more than one of the original tags were recoded into one 
and the same new tag. We applied different recodings, thus obtaining English corpora, lexicons 
10 and guessers with reduced tag sets of 45, 36, 27, 18 and 9 tags respectively. 



Transducer 
or 
HMM 


Accuracy 

test corp. 
in% 


Tagging speed 
in words/second 


Transducer size 


Creation 

tone 
on Uftra2 




nurrtof 
stales 


nunuof 
arcs 




onUltra2 


onSparc20 


HMM 


97.07 


3 358 


1363 








b-FST O=0.a=0) 


94.47 


25 521 


11 815 


1 


119 


3 sec 


b-FST (p=1,ce=0) 


95.60 


25 521 


12038 


28 


3332 


4 sec 


b-FST (P=2,o=0) 


97.71 


25 521 


11 193 


1642 


195398 


32ntin 


b-FST (p=0,o=1) 


95.26 


17244 


9 969 


137 


14074 


5 sec 


b-FST (P=0,a=2) 


95.37 


19939 


9 969 


3685 


280545 


3min 


b-FST ({J=1 ,0=1) 


•96.78 


16 790 


8 986 


1748 


192275 


19 sec 


b-FST (p=2,a=1) 


•97.06 


22787 


11 000 


19878 


1911277 


61 min 



Language 


English 


Corpora 


19 944 words for HMM training. 19 934 words for test 


Tag set 


27 tags, 119 classes 


• 


Multiple, Le. ambiguous tagging results: Only first result retained 


Types of FST (Finite State Transducers) 


b(p=2.a=1> 


b-type transducer with look-back of 2 and look-ahead of 1 


Computers 




Uttra2 


1 CPU, 512 MBytes physical RAM, 1.4 GBytes virtual RAM 


Sparc20 


1 CPU, 192 MBytes physical RAM, 827 MBytes virtual RAM 



Table 3: Accuracy, tagging speed and size of some HMM transducers 
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Table 3 compares b-type transducers with different length of look-back arid look-ahnd 
for a tag set of 27 tags. The highest accuracy (97.06 %) could be obtained with an b-type FST 
with p=2 and a=1. This b-type FST produced in some cases ambiguous tagging results. Then 
only the first result found was retained. 



Tagging accuracy with tag sets of different sizes (in %) 



Transducer or 
HMM 


74 tags 
297 ds. 


45 tags 
214cls. 


38 tans 
181 ds. 


07 tarie 
lay 

119ds. 


in tans 
to uuja 

97 ds. 


67 ds. 


HMM 


96.78 


96.92 


97.35 


97.07 


96.73 


95.76 


b-FST (P=0.<*=0) 


8&53 


83.71 


87.21 


94.47 


94.24 


93.86 


b-FST (0=1 ,cc=Q) 


94.19 


94.09 


95.16 


95.60 


95.17 


94.14 


b-FST 0=2,ot=0) 




94.28 


95.32 


95,71 


95.31 


94.22 


b-FST 0=0 t a=1) 


92.79 


92.47 


93.69 


95^6 


95.19 


94.64 


b-FST (p=0 t a=2) 


93.46 


92.77 


93.92 


9537 


95.30 


94.80 


b-FST (p=1,a=1) 


•94.94 


•95.14 


•95.78 


•96.78 


•96.59 


•9536 


b-FST 0=2,0=1) 






•97.34 


•97.06 


•96.73 


•95.73 


b-FST 0=3.cc=1) 












95.76 











Language English 

Corpora 1 9 944 words for HMM training, 1 9 934 words for test 

Types of FST (Rnite State Transducers) see Table 3 

. Multiple, Le. ambiguous tagging results: Only first result retained 
j j Transducer could not be computed for reasons of size. 



Table 4: Tagging accuracy with tags sets of different sizes 

10 Table 4 shows the tagging accuracy of different b-type transducers with tag sets of 

different sizes. To get results that are almost equivalent to those of an HMM, a b-type FST 
needs at least a look-back of p=2 and a look-ahead of a=l or vice versa. For reasons of 
size, this kind of FST could only be computed for the tag sets with 36 tags or less. A fa- 
type FST with p=3 and a=1 couid only be computed for the tag set with 9 tags. This FST 

15 gave exactly the same tagging results as the underlying HMM. 
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Annex: Regular Expression Operators of the XEROX Finite State Calculus 

Below, a and b designate symbols, A and B designate languages, and R and Q designate 
relations between two languages. More details on the following operators and pointers to fin'rte- 
5 state literature can be found in http://www.rxrc.xerox.com/research/mltt/fst 



$a Contains. Set of strings containing at least one occurrence of a string from A as a 

substring. 

-A Complement (negation). AH strings except those from A. 

\a Term complement Any symbol other than a. 

A* Kleene star. Zero or more times A concatenated with itself. 

A^n A n times. Language A concatenated n times with itself. 

A + Kleene plus. One or more times A concatenated with ftseff. 

a -> b Replace. Relation where every a on the upper side gets mapped to a b on the lower 
side. 

a <- b Inverse replace. Relation where every b on the lower side gets mapped to an a on 
the upper side. 

Symbol pair with a on the upper and b on the lower side. 
r.u Upper language of the relation R. 

R-l Lower language of the relation R. 

R-i Inverse relation where the upper and the lower language are exchanged with 

respect to R. 

A B Concatenation of aO strings of A with all strings of B. 

a | B Union of the languages A and B. 

a & B Intersection of the languages A and B. 

A - B Relative complement (minus). All strings of the language A that are not in B. 
A .x. B Cross Product (Cartesian product) of the languages A and B. 
R -o. Q Composition of the relations R and GL 
0 or [ ] Empty string (epsflon). 

? Any symbol in the known alphabet and its extensions. 
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CLAIMS: 

1 . A method carried out in a text processing system, for generating a text tagging finite state 
transducer (FST) for a predetermined language, the finite state transducer encoding along a 
5 plurality of arcs sets of ordered pairs of upper and lower labels, comprising: 

(a) providing a set of tags, the or each tag being a part-of-speech designator for the 
language, and a set of classes, the or each class being ambiguity classes, defining 
groups of possible tags, for words in the language, 

(b) providing a plurality of states, including an initial state and, for each class-tag pair, a 
10 further state labelled as the respective class-tag pair, and 

(c) for each state, for each class, creating an arc between the state and a destination 
state labelled with a class-tag pair, the class tag pair being the most probable for 
the class, the arc so added having the class as its upper label and a tag of that 
class as its lower label. 



15 



20 



25 



2. The method of claim 1, wherein (c) comprises, where the state is the initial state, (d) 
determining the most probable class tag pair using 

argmaxp^,^) =n(t ik yb(c i \t ik ) 

where tc is initial probability, and b is class probability. 

3. The method of claim 1 or 2, wherein (c) comprises, where the state is not the initial state, 
(d *) determining the most probable class tag pair using 

where a is transition probability, and b is class probability. 

4. The method of claim 1, 2 or 3, further comprising minimising and determmtsing the FST 
generated in steps (a)- (c). 



5. A method carried out in a text processing system, for generating a text tagging finite state 
30 transducer (FST) for a predetermined language, the finite state transducer encoding along a 
plurality of arcs sets of ordered pairs of upper and lower labels, comprising: 

(a) providing an incomplete set of class sequences, the class sequences being 
sequences of classes, the or each class being ambiguity classes, defining groups 
of possible tags, for words in the language, 
35 (b) tagging the class sequences using a Hidden Markov Model, to produce an 

incomplete set of class and tag sequences of a first type, 
(c) providing a complete set of class and tag sequences of a second type. 
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(d) combining the set of class and tag sequences ot a nrst type with the set o* clas? 
and tag sequences of a second type, and 

(e) building a FST using the combination obtained in step (d). 

5 6. The method of claims 5, wherein step (a) comprises: 
(a1 ) providing a set of tags and a set of classes, and 

(a2) creating from said set of tags and set of classes, all possible class sequences up to 
a predetermined length. 

10 7. The method of claim 5, wherein step (a) comprises: 
(a! 1 ) providing an untagged text corpus, and 
(a?) extracting all class sequences from the untagged text corpus. 

8. The method of claim 5, 6 or 7, wherein step 6 comprises: 

15 (d ) providing a FST generated according to the method of any of claims 1 to 4, 

(c2) extracting all class sequences from the FST provided in 

9. A method carried out in a text processing system, for generating a text tagging finite-state 
transducer (FST) for a predetermined language, the FST encoding along a plurality of arcs sets of 

20 ordered pairs of upper and lower labels, comprising: 

(a) providing a set of tags, the or each tag being a part-of-speech designator for the 
language, and a set of classes, the or each class being an ambiguity class, defining 
groups of possible tags, for the language, 

(b) creating from the set of tags and the set of classes a set of subsequences of 
25 ambiguity classes, said set of subsequences comprising all possible subsequences 

for a predetermined look-back value and look-ahead value, 

(c) using a Hidden Markov Model, tagging said set of subsequences to generate a set 
of tagged subsequences, 

(d) performing a union operation, followed by a Kleene star operation, on the set of 
30 tagged subsequences, to generate a preliminary tagger model, said preliminary 

tagger model being a sequence in which the tagged subsequences may appear in 
any order or in any number, 

(e) composing a plurality of constraints with the preliminary tagger model to generate 
the tagging FST, said constraints including constraints for tags, classes and 

35 sentence boundaries, and deleting all symbols expressing constraints. 

10. A method carried out in a text processing system, fore tagging an untagged text corpus, 
comprising: 

(a) reading, in sequence, the or each word of the text corpus, 
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(b) for the or each word, looking up tie word using a lexical .esouxe, to obtain the 
word labelled with its dass, 

(c) if a sentence end token has not been read, repeating steps (a) and (b), the 
sentence end token defining the end of the current sentence, 

5 (d) if a sentence end token has been read, applying the FST obtainable by the method 

of claims 1 , 5 or 9 to the class sequence corresponding to the current sentence, to 
generate a tag sequence, the tag sequence comprising a sequence of tags, 
(e) outputting a tagged form of the current sentence, the tagged form comprising the or 
each word of the current sentence and, appended thereto, a respective tag from 
10 said tag sequence, and 

(0 if the end erf the text corpus has not been reached, repeating steps (a)-(e). 

11. A text processing system when suitably programmed for carrying out the method of any of 
the preceding daims, the system comprising a processor and memory, the processor being 

15 operable with said memory for executing instructions corresponding to steps of any of said 
methods. 

12. A recordable medum having recorded thereon digital data defining instructions for 
execution by a processor and corresponding the steps of the methods of any of claims 1 to 1 0. 

20 
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